Papers with selection strategy
On Training Instance Selection for Few-Shot Neural Text Generation (2021.acl-short)
Copied to clipboard
| Challenge: | Pretraining large neural networks with a language modeling objective has led to dramatic improvements in text generation. |
| Approach: | They propose a selection strategy to select few-shot training instances based on unlabeled data to identify the most worthwhile data points that should be annotated under some budget of labeling cost. |
| Outcome: | The proposed strategy outperforms random sampling on three text generation tasks. |
Impact of Training Instance Selection on Domain-Specific Entity Extraction using BERT (2022.naacl-srw)
Copied to clipboard
| Challenge: | Named entity recognition (NER) tasks require a large number of training examples and handcrafted features. |
| Approach: | They propose to fine-tune pre-trained language models such as BERT to achieve up to 80% F1 when fine- tuned on only 70 training examples. |
| Outcome: | The proposed model achieves 80% F1 when fine-tuned on only 70 training examples, especially on biomedical domain. |
Neural Document Summarization by Jointly Learning to Score and Select Sentences (P18-1)
Copied to clipboard
| Challenge: | Sentence scoring and sentence selection are two main steps in extractive document summarization systems. |
| Approach: | They propose an end-to-end neural network framework for extractive document summarization by jointly learning to score and select sentences. |
| Outcome: | The proposed framework outperforms the state-of-the-art summarization models on the CNN/Daily Mail dataset. |
EDU-level Extractive Summarization with Varying Summary Lengths (2023.findings-eacl)
Copied to clipboard
| Challenge: | Existing studies on extractive summarization use finer-grained elementary discourse units . few studies exploited finer grained EDUs with little analysis and justification for the extractive unit selection . |
| Approach: | They propose an extractive model with Varying summary lengths that extracts fixed top-k salient sentences from the document as a summary. |
| Outcome: | The proposed model performs better on ROUGE scores than state-of-the-art models. |
Sentence Selection Strategies for Distilling Word Embeddings from BERT (2022.lrec-1)
Copied to clipboard
| Challenge: | Using language models to learn word embeddings is a key feature of transformer-based language models. |
| Approach: | They propose to use language models to learn high-quality word vectors from as few as 5 to 10 sentences with a careful selection strategy. |
| Outcome: | The proposed strategies can learn high-quality word vectors from as few as 5 to 10 sentences. |
TR-BERT: Dynamic Token Reduction for Accelerating BERT Inference (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing pre-trained language models (PLMs) are expensive in inference, making them impractical in resource-limited real-world applications. |
| Approach: | They propose a dynamic token reduction approach to accelerate PLMs' inference by adapting the layer number of each token to avoid redundant calculation. |
| Outcome: | The proposed approach speeds up BERT by 2-5 times and improves performance in long-text tasks with less computation. |
What are the Desired Characteristics of Calibration Sets? Identifying Correlates on Long Form Scientific Summarization (2023.acl-long)
Copied to clipboard
Griffin Adams, Bichlien Nguyen, Jake Smith, Yingce Xia, Shufang Xie, Anna Ostropolets, Budhaditya Deb, Yuan-Jyue Chen, Tristan Naumann, Noémie Elhadad
| Challenge: | Summarization models are trained to maximize the likelihood of a single reference (MLE) but little is known about why one setup is more effective than another . |
| Approach: | They add a calibration step which exposes a model to its own ranked outputs to improve relevance or contrasts positive and negative sets to improve faithfulness. |
| Outcome: | The proposed calibration step can unlock large gains in relevance or faithfulness. |
Transfer and Active Learning for Dissonance Detection: Addressing the Rare-Class Challenge (2023.acl-long)
Copied to clipboard
Vasudha Varadarajan, Swanie Juhng, Syeda Mahwish, Xiaoran Liu, Jonah Luby, Christian Luhmann, H. Andrew Schwartz
| Challenge: | Active learning has been proposed to alleviate data acquisition challenges for rare-class tasks when the class label is very infrequent (e.g., 5% of samples). |
| Approach: | They propose to use transformers to train models on closely related tasks and evaluate acquisition strategies, including a proposed probability-of-rare-class approach to dissonance detection. |
| Outcome: | The proposed method improves model accuracy while iterative transfer-learning does not improve cold-start performance. |
Towards Multiple References Era – Addressing Data Leakage and Limited Reference Diversity in Machine Translation Evaluation (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent research shows a weak correlation between n-gram-based metrics and human evaluations in machine translation tasks. |
| Approach: | They propose to use multiple references generated by LLMs to improve alignment between automatic metrics and human evaluations. |
| Outcome: | The proposed approach improves the alignment between automatic metrics and human evaluations on the WMT22 benchmark with 4 languages and achieves a maximum accuracy gain of 9.5%. |